From Adaptive Linear to Information Filtering

Adaptive signal processing theory was born and has lived by exclusively exploiting the mean square error criterion. When we think of the goal of least squares without restrictions of Gaussianity, one has to wonder why an information theoretic error criterion is not utilized instead. After all, the goal of adaptive filtering should be to find the linear projection that best captures the information in the desired response. In this paper we summarize our efforts to extend adaptive linear filtering to information filtering. We briefly review Renyi’s entropy definition, Parzen windows and put them together in a framework to estimate entropy directly from samples (nonparametric). Once this criterion is developed we can train linear or nonlinear adaptive networks for entropy maximization or minimization. We present results on the properties of the Renyi’s nonparametric entropy estimator, and show how it performs in chaotic time series prediction.